PROTOCOL: Critical appraisal of methodological quality and reporting items of systematic reviews with meta‐analysis in evidence‐based social science in China: A systematic review

Abstract This is the protocol for a Campbell systematic review. The objectives are as follows: (1) To evaluate the reporting quality of systematic reviews published in Chinese social science journals against the PRISMA and MOOSE standards; (2) To evaluate the methodology quality of systematic reviews published in Chinese social science journals against the AMSTAR‐2 and DART standards; and (3) To analyze other characteristics of systematic reviews published in Chinese social science journals using content analysis.


| BACKGROUND
Evidence-based practice has gradually developed, and more systematic reviews and meta-analyses have been conducted in the field of social sciences in China. Systematic reviews aim to summarize "the best available research on a specific question by synthesizing the results of several studies." They use transparent procedures to search, evaluate, and synthesize the results of relevant research whilst minimizing bias. Because of this integrative effect, systematic reviews can provide more valuable references for policy-making and practice than individual studies. Therefore, for example, the Oxford Centre of Evidence-Based Medicine recommended systematic review/meta-analysis as Level A evidence (Level 1 evidence) in the evidence classification (Gates & March, 2016). The Growing What Works movement in the United Kingdom bases its evidence-based decision-making products on systematic reviews (Gough et al., 2018).
High-quality systematic reviews and meta-analyses can provide effective and scientific evidence for decision-makers, and they are also the primary source of information for researchers to quickly grasp the current progress of a research problem (Oliver & Dickson, 2015). However, the reporting quality of a systematic review depends on its methodological rigor and the clarity of the research report. Besides, differences in methods may lead to completely opposite conclusions of systematic reviews on the same research topic (Schalken & Rietbergen, 2017;Vrieze, 2018) (Mulrow, 1987), Meanwhile, Sacks et al. (1987) evaluated the reporting adequacy of 83 English-language meta-analyses of randomized controlled trials in the medical field that published from January 1966 through October 1986, involving in six areas: study design, combinability, control of bias, statistical analysis, sensitivity analysis, and application of results. The results showed that the quality of the reports from these reviews was low, with only 1-14 items of 83 meta-analyses being fully reported (Sacks et al., 1987).
After 10 years, Sacks et al. (1996) updated the study finding that the situation had hardly improved (Sacks et al., 1996).
To improve the quality of meta-analysis, Moher et al. (1999) issued a guideline, named Quality of Reporting of Meta-Analysis (QUOROM), focusing on the report quality of meta-analysis on Randomized controlled trials (Moher et al., 1999). In 2009, they revised QUOROM guidelines and renamed them Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), which also considered the quality of reporting on systematic reviews. With the advances in systematic review methodology and terminology, Page et al. (2021) developed the PRISMA 2020 statement for reporting systematic reviews based on 60 documents with reporting guidelines for systematic reviews to generate suggested modifications to the PRISMA 2009 statement in 2021 (Page et al., 2021). Meanwhile, the number of published metaanalyses concerning observational studies in health has increased substantially, Stroup et al. (2000) held a workshop in Atlanta Ga, in April 1997, and proposed a reporting checklist for Meta-analysis of Observational Studies in Epidemiology (MOOSE) to examine the reporting of meta-analyses of observational studies (Stroup et al., 2000).
In addition to the reporting checklist, Shea et al. (2009) developed a measurement tool to assess systematic reviews (AMSTAR) to evaluate the methodological quality of systematic reviews on randomized controlled trials (Shea et al., 2009). After receiving comments and feedback, the AMSTAR group revised AMSTAR and released AMSTAR-2 in September 2017 (Shea et al., 2017), which also included non-randomized studies of interventions (NRSI). In 2015, Diekemper et al. (2015) developed a Documentation and Appraisal Review Tool (DART) for systematic reviews, which explicitly included a quality review for biases specific to observational studies.
After these guidelines were released, more studies were conducted to explore the methodology and reporting quality of systematic reviews and meta-analyses in medical research, such as substance abuse (Kim et al., 2021), pediatrics (Bo et al., 2020), nursing (Jin et al., 2014), orthopedics (Gagnier & Kellam, 2013), and etc.
In recent years, systematic reviews and meta-analyses have been increasing in the field of social sciences. Social science is taken to mean any branch of academic study or science that deals with human behavior in its social and cultural aspects. It is mainly focused on the scientific study of human society and social relationships. Some researchers have already assessed the quality of systematic reviews in social science. For example, Kogut et al. (2019) examined the reporting quality of mathematics education systematic review with 40 reviews, they found deficiencies in search processes and reporting of search methods (Kogut et al., 2019). Wang et al. (2021) examined the reporting quality of 96 Campbell Systematic Reviews, finding that fewer than half (42%) were of high quality, but that quality had risen since standards were introduced (Wang et al., 2021).
In China, reviews are being undertaken in areas including Marxist Theory Studies, Management Science, Philosophy, Religion Studies, Linguistics, Law, Education, Economics, Geography, Ethnography, and Cultural Studies, Archaeology, History, Psychology, Sociology, Journalism, and Communication Studies, Political Science, Library Information Science, Sport, and Art.
Meanwhile, Chinese researchers introduced the critical appraisal methods with Chinese versions of critical appraisal tools Li et al., 2009;Tao et al., 2018;Tian et al., 2015;Xiong & Chen, 2011;Zhan, 2010;Zhang et al., 2015). These tools have been widely used in systematic reviews in medical fields , and analyzed the methodology and reporting quality of Chinese reviews. Tian et al. (2017) compared the methodological and reporting quality of 100 systematic reviews by authors from China and those from the United States and found them to be of similar quality . In 2022, Bai et al. randomly selected 200 Chinese systematic reviews in the social science field published from 2000 to 2019 in the Chinese Social Sciences Citation Index (CSSCI) database. They examined the methodological and reporting quality of these reviews and suggested that the quality of the systematic reviews was below the average level (Bai et al., 2022). However, the data source they searched was the CSSCI database which covers 500 of 2700 Chinese academic journals of social sciences, thus, it is reasonable to doubt possible selection bias in their review. Therefore, the present study aims to evaluate whether and to what extent reporting and methodology standards are met in the systematic reviews of social science in China and to assess the applicability of these tools in the Chinese context with content analysis.

| OBJECTIVES
The present review includes three objectives: 1. To evaluate the reporting quality of systematic reviews published in Chinese social science journals against the PRISMA and MOOSE standards.
2. To evaluate the methodology quality of systematic reviews published in Chinese social science journals against the AMSTAR-2 and DART standards.
3. To analyze other characteristics of systematic reviews published in Chinese social science journals using content analysis.

| Criteria for considering studies for this study
Completed systematic reviews and meta-analyses published in Chinese journals between January 2009 (when PRISMA was released) and January 2022. We will include intervention reviews and observational systematic reviews with meta-analysis in social Identify the report as a systematic review.

Rationale 3
Describe the rationale for the review in the context of existing knowledge.
Objectives 4 Provide an explicit statement of the objective(s) or question(s) the review addresses.

Eligibility criteria 5
Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses.
Information sources 6 Specify all databases, registers, websites, organizations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted.
Search strategy 7 Present the full search strategies for all databases, registers and websites, including any filters and limits used.
Selection process 8 Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process.
Data collection process 9 Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process.

Data items 10a
List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g. for all measures, time points, analyses), and if not, the methods used to decide which results to collect. 10b List and define all other variables for which data were sought (e.g. participant and intervention characteristics, funding sources). Describe any assumptions made about any missing or unclear information.
Study risk of bias assessment 11 Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process.
Effect measures 12 Specify for each outcome the effect measure(s) (e.g. risk ratio, mean difference) used in the synthesis or presentation of results.

Synthesis methods 13a
Describe the processes used to decide which studies were eligible for each synthesis 13b Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data conversions. 13c Describe any methods used to tabulate or visually display results of individual studies and syntheses. 13d Describe any methods used to synthesize results and provide a rationale for the choice(s). If metaanalysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used. 13e Describe any methods used to explore possible causes of heterogeneity among study results. 13f Describe any sensitivity analyses conducted to assess robustness of the synthesized results.
Reporting bias assessment 14 Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases).

Certainty assessment 15
Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.

Study selection 16a
Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram.
16b Cite studies that met many but not all inclusion criteria ('near-misses') and explain why they were excluded.

Study characteristics 17
Cite each included study and present its characteristics.

Risk of bias in studies 18
Present assessments of risk of bias for each included study.

Results of individual studies 19
For all outcomes, present, for each study: (a) summary statistics for each group (where appropriate) and (b) an effect estimates and its precision (e.g. confidence/credible interval), ideally using structured tables or plots.

Results of syntheses 20a
For each synthesis, briefly summaries the characteristics and risk of bias among contributing studies. 20b Present results of all statistical syntheses conducted. If meta-analysis was done, present for each the summary estimate and its precision (e.g. confidence/credible interval) and measures of statistical heterogeneity. If comparing groups, describe the direction of the effect. 20c Present results of all investigations of possible causes of heterogeneity among study results. 20d Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results.

Reporting biases 21
Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed.

Certainty of evidence 22
Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed.

Discussion 23a
Provide a general interpretation of the results in the context of other evidence. 23b Discuss any limitations of the evidence included in the review. 23c Discuss any limitations of the review processes used.
23d Discuss implications of the results for practice, policy, and future research.

Registration and protocol 24a
Provide registration information for the review, including register name and registration number, or state that the review was not registered.

24b
Indicate where the review protocol can be accessed, or state that a protocol was not prepared. 24c Describe and explain any amendments to information provided at registration or in the protocol.

Support 25
Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review.

Competing interests 26
Declare any competing interests of review authors.

Availability of data, code and other materials 27
Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review.

| Data collection and analysis Selection of studies
The selection of studies will be performed independently by two reviewers (Mina Ma and Minyan Yang) in Rayyan. All titles and abstracts of the records identified after retrieval will be screened, the potentially relevant references will be located with full text, and the systematic reviews that meet our criteria will be included for further analysis. Any discrepancies between the two reviewers will be resolved by consensus with another reviewer involved (Zhipeng Wei). The whole process of study screening will be reported as in the PRISMA guideline (Moher et al., 2009).

| Data extraction and management
Information extraction and coding will consist of two parts. The first is the information of publication, including the first author, title, publication year, and source of literature. The other part is the characteristics of the study content, including nine sections and 38 items (nine sections include study field, study design, title, abstract, introduction, method, result, discussion, and other information). This process will be performed by two reviewers independently (Xin Xing and Jieyun Li), and any questions will be resolved with the third author (Wenjie Zhou). Before the formal extraction, three rounds of piloting coding with 15-20 included studies will be conducted by the three authors independently using Microsoft EXCEL2019 until they reach an agreement on the extraction items. The extraction items are shown in Table 1.

| Quality appraisal
The reporting and methodological quality of intervention systematic reviews will be assessed using PRISMA2020 guideline and AMSTAR-2 and observational systematic reviews will be evaluated with the MOOSE checklist DART tool. Each item of the assessment checklist will be performed in EXCEL, and the overall confidence in the results of the reviews evaluated by AMSTAR-2 will be rated automatically on AMSTAR web. This process will be T A B L E 3 Items of PRISMA abstract Section/Topic Item Checklist item TITLE Title 1 Identify the report as a systematic review.

Objectives 2
Provide an explicit statement of the main objective(s) or question(s) the review addresses.

Eligibility criteria 3
Specify the inclusion and exclusion criteria for the review.
Information sources 4 Specify the information sources (e.g. databases, registers) used to identify studies and the date when each was last searched.
Risk of bias 5 Specify the methods used to assess risk of bias in the included studies.
Synthesis of results 6 Specify the methods used to present and synthesize results.

Included studies 7
Give the total number of included studies and participants and summaries relevant characteristics of studies.
Synthesis of results 8 Present results for main outcomes, preferably indicating the number of included studies and participants for each. If metanalysis was done, report the summary estimate and confidence/credible interval. If comparing groups, indicate the direction of the effect (i.e. which group is favored).

Limitations of evidence 9
Provide a brief summary of the limitations of the evidence included in the review (e.g. study risk of bias, inconsistency and imprecision).
Interpretation 10 Provide a general interpretation of the results and important implications.

Funding 11
Specify the primary source of funding for the review.

Registration 12
Provide the register name and registration number. 19. Documentation of how data were classified and coded (e.g., multiple raters, blinding, and interrater reliability) 20. Assessment of confounding (e.g., comparability of cases and controls in studies where appropriate 21. Assessment of study quality, including blinding of quality assessors; stratification or regression on possible predictors of study results

Assessment of heterogeneity
23. Description of statistical methods (e.g., complete description of fixed or random effects models, justification of whether the chosen models account for predictors of study results, dose-response models, or cumulative meta-analysis) in sufficient detail to be replicated

Provision of appropriate tables and graphics
Reporting of results should include 25. Graphic summarizing individual study estimates and overall estimate 26. Table giving descriptive information for each study included 27. Results of sensitivity testing (e.g., subgroup analysis)

Indication of statistical uncertainty of findings
Reporting of discussion should include 29. Quantitative assessment of bias (e.g., publication bias) 30. Justification for exclusion (e.g., exclusion of non-English-language citations) 31. Assessment of quality of included studies conducted independently by two authors (Xin Xing and Jieyun Li), and disagreements between coders will be resolved by discussions with another author (Wenjie Zhou). Before the formal assessment, three rounds of piloting with 15-20 included reviews will be performed to test the consistency of raters until it reaches over 95%. The items of each tool are shown in Tables 2-6.

| Data synthesis
We will use the new PRISMA guideline and MOOSE checklist to reflect the reporting quality of interventional and observational systematic reviews, and descriptive statistics (frequencies and percentages) will be used to describe reporting quality of systematic reviews. We will Did the authors describe the methods they used to combine/synthesize the results of the relevant studies (to reach a conclusion) and were the methods used appropriate for the review question(s)?
a Methods were reported clearly enough to allow for replication. The overview included some assessment of the qualitative and quantitative heterogeneity of the study results and the results were appropriately combined/ synthesized. For meta-analyses, an accepted pooling method (i.e., more than simple addition) was used. Or the authors state that the evidence is conflicting and that they can't combine/synthesize the results

Yes b
The methods were reported clearly enough to allow for replication but they were not combined appropriately Partially c Not described or cannot tell No 12 Did the authors perform sensitivity analyses on any changes in protocol, assumptions, and study selection? (For example, using sensitivity analysis to compare results from fixed effects and random effects models) a Sensitivity analyses were used when appropriate on all changes in a priori design Yes b Sensitivity analyses were only used on some changes in a priori design Partially c Not described or cannot tell No 13 Are the conclusions of the authors supported by the reported data with consideration of the overall quality of that data?
a The conclusions are supported by the reported data and reflect both the scientific quality of the studies and the risk of bias in the data obtained from those studies Yes b The authors failed to consider study quality and/or their conclusions were not supported by the data, or cannot tell No

14
Were conflicts of interest stated and were individuals excluded from the review if they reported substantial financial and intellectual conflicts of interests? adopt AMSTAR-2 and DART to evaluate the methodology quality of interventional and observational systematic reviews respectively, and the results will be shown with the percentages of each grade (high, moderate, low, and critical low with AMSTAR-2; good, fair, and poor with DART). The descriptive statistics (frequencies and percentages) will be used to describe reporting characteristics of systematic reviews based on the content analysis information.

| Planed moderators
Subgroup analyses with stratification analysis will be conducted to explore the potential difference in reporting and methodology quality depending on the source of literature and research field. The regression analysis will be performed to evaluate the differences in quality.
• Source of literature. According to quantitative indexes such as the influence factors of periodicals, the total number of citations, and the opinions of experts in various disciplines, the China Social Science Research and Evaluation Centre of Nanjing University developed a Chinese Social Science Citation Index (CSSCI) which included the high influence of Chinese academic journals in social science filed. We plan to compare the quality of reviews appearing in journals indexed in CSSCI and those that are not in CSSCI.
• Research field. We will group the reviews into 19 fields, including Marxist Theory Studies, Management Science, Philosophy, Religion Studies, Linguistics, Law, Education, Economics, Geography, Ethnography and Cultural Studies, Archaeology, History, Psychology Sociology, Journalism and Communication Studies, Political science, Library Information Science, Sport, and Art. The subgroup analysis will be used to variation in quality between different fields.
• Publication year. After the systematic review and meta-analysis guidelines were released, Chinese researchers introduced the Chinese version in 2009. We will conduct a regression analysis with the publication year as the independent variable and the quality of the reviews as the dependent variable to find the change in the reporting and methodology quality of Chinese reviews in social science.

CONTRIBUTIONS OF AUTHORS
Guo L. P. drafted the protocol, and all authors reviewed the draft and approved the final version.

DECLARATIONS OF INTEREST
All authors declare no potential interest.

SOURCES OF SUPPORT
Internal sources